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TUTORIAL 


DEVELOPMENT  OF  RISK  MANAGEMENT 
DEFENSE  EXTENSIONS  TO  THE 
PMI  PROJEa  MANAGEMENT 
BODY  OF  KNOWLEDGE 


Edmund  H,  Conroiv,  Ph,D, 

This  article  describes  the  risk  management  defense  extensions  to  the  2000 
Project  Management  Institute  (PMI)  Project  Management  Body  of  Knowledge 
(2000  PMBOK®  Guide).  The  Department  of  Defense  (DoD)  Draft  Extension 
was  developed  to  provide  recommended  tailoring  of  the  2000  PMBOK®  Guide 
to  Department  of  Defense-specific  applications.  The  focus  of  this  article  is  on 
Department  of  Defense-specific  tailoring  associated  with  risk  management 
information  that  appears  in  Chapter  1 1  of  the  2000  PMBOK®  Guide,  including 
key  supplemental  information  and  enhancements. 


The  concept  of  a  Department  of  De¬ 
fense  (DoD)  Extension  to  the  Project 
Management  Body  of  Knowledge 
(PMBOK®)  Guide  has  its  beginnings  as 
early  as  the  1992  Project  Management 
Institute  (PMI)  Symposium  in  Pittshurgh, 
Pennsylvania.  Several  PMI  members  — 
all  members  of  the  PMI  Aerospace/De¬ 
fense  Specific  Interest  Group  (A&D  SIG) 
—  agreed  that  a  document  supplement¬ 
ing  the  current  PMBOK®  Guide,  with  spe¬ 
cific  information  on  Defense  Acquisition, 
was  necessary  for  both  defense  contrac¬ 
tors  and  foreign  governments  that  pro¬ 
cure  defense  systems.  Besides  supple¬ 
mental  DoD  process  information  for  each 
of  the  eight  main  PMBOK®  knowledge 


areas  (including  risk  management),  five 
additional  “defense  intensive”  areas  were 
identified  for  inclusion  in  any  extension 
to  the  PMBOK®  Guide.  These  five  addi¬ 
tional  areas  are  Systems  Engineering 
Management,  Software  Development 
Management,  Test  and  Evaluation  Man¬ 
agement,  Eogistics  Management,  and 
Manufacturing  Management.  In  line  with 
current  DoD  policy,  each  of  these  five  ar¬ 
eas  —  and  other  PMBOK®  Guide  supple¬ 
mental  material  as  well  —  are  primarily 
composed  of  commercial  practices;  these 
practices  have  become  an  integral  part  of 
DoD  acquisition  processes. 

In  1996,  the  PMI  Standards  Committee 
initiated  a  project  to  develop  the  process 
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to  implement  the  provision  for  exten¬ 
sions.  The  final  report  and  proposed  pro¬ 
cess  were  formally  transmitted  to  the  Di¬ 
rector  of  Standards  in  March  1997;  this 
event  launched  the  formal  development 
hy  the  Defense  Systems  Management 
College/Defense  Acquisition  University 
(DSMC/DAU)  of  the  Defense  Extension 
to  the  PMBOK®  Guide.  At  the  same  time, 
it  was  agreed  between  PMI  and  DSMC/ 
DAU  that  the  extension 
would  focus  on  United 
States  DoD  concepts, 
processes,  and  proce¬ 
dures,  with  appropriate 
reference  to  the  acqui¬ 
sition  systems  of  allied 
foreign  nations.  The 
goal  was  to  develop  a 
PMI  standard  that  could 
he  used  hy  defense  con¬ 
tractors  and  foreign  na¬ 
tions  alike.  The  DoD 
Extension  to  the 
PMBOK®  Guide  began 
to  take  shape  in  1999, 
when  the  work  of  23  government  and  De¬ 
fense  industry  contributors  coalesced  into 
a  coherent  usable  document. 

At  that  time  it  was  anticipated  that  the 
extension  would  be  published  “in  the  pub¬ 
lic  domain”;  that  PMFs  copyright  for  the 
PMBOK®  Guide  needed  to  be  protected, 
and  that  the  extension  —  as  a  PMBOK® 
Guide  derivative  work  —  also  needed  to 
incorporate  protection  of  PMI  intellectual 
property. 

At  the  time  that  this  article  is  being 
written,  the  DoD  Extension  is  a  draft 
document,  i.e.,  the  DoD  Draft  Extension, 
that  has  been  reviewed  by  the  PMI  mem¬ 
bership.  Eollowing  completion  of  the  re¬ 
view  process,  it  is  envisioned  that  the  draft 
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will  become  an  accepted  PMI  standard. 
While  this  article  focuses  on  the  Risk  Man¬ 
agement  part  of  the  DoD  Extension,  it  is 
important  to  keep  a  full  perspective  on  all 
the  parts  that  make  up  the  Extension,  since 
they  are  all  important  to  the  “art”  of  DoD 
aerospace/defense  program  management. 

The  risk  management  process  pre¬ 
sented  in  the  Project  Management 
PMBOK®  Guide,  generally  applies  to 
DoD  acquisition  programs.  The  DoD 
Draft  Extension  PMBOK®  Guide  provides 
supplemental  information  needed  for  the 
risk  management  of  defense  systems,  as 
well  as  enhancements  to  the  2000 
PMBOK®  Guide  material. 

Information  is  provided  in  the  DoD 
Draft  Extension  for  several  supplemental 
topics,  including: 

•  DoD  risk  management  policy; 

•  A  summary  of  DoD  risk  management 
principles  and  lessons  learned; 

•  DoD  risk  management  structure; 

•  DoD  risk  management  definitions  and 
the  Risk  Management  Process  Model; 

•  Organizational  and  behavioral  consid¬ 
erations  for  implementing  risk  man¬ 
agement; 

•  The  performance  dimension  of  con¬ 
sequence  of  occurrence; 

•  The  performance  dimension  of  Monte 
Carlo  simulation  modeling; 

•  A  structured  approach  for  developing 
a  risk  handling  strategy.  (Defense 
Acquisition  University  (DAU),  2002a). 
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This  article  briefly  addresses  a  few 
of  these  supplemental  items. 

DoD  Risk  Management  Structure 
AND  Process  Model 


In  the  DoD  risk  management  process 
structure,  given  in  Figure  1  (DAU, 
2002b),  there  are  four  process  steps, 
with  risk  assessment  further  broken  down 
into  risk  identification  and  risk  analysis. 
This  process  structure  is  similar  to  that 
given  in  the  2000  PMBOK®  Guide 
(Project  Management  Institute  [PMI], 
2000),  except  for  the  following  consid¬ 
erations.  First,  risk  analysis  is  split  into 
qualitative  and  quantitative  risk  analysis 
process  steps  in  the  2000  PMBOK® 
Guide,  whereas  in  the  DoD  Draft  Exten¬ 
sion  it  is  treated  as  a  single  process  step. 
The  rationale  for  the  DoD  approach  is 
that  many  of  the  same  inputs,  resources, 
and  use  of  outputs  exist  for  both  qualita¬ 
tive  and  quantitative  risk  analysis,  and 


some  potential  methodologies  blur  the 
boundary  between  these  forms  of  risk 
analysis  (e.g.,  the  use  of  probability  tables 
estimated  from  a  statistical  analysis  of 
survey  results).  Second,  the  DoD  Draft 
Extension  emphasizes  the  feedback  term 
present  from  risk  monitoring  (as  shown 
in  Eigure  1)  to  the  other  process  steps, 
which  is  not  illustrated  in  the  2000 
PMBOK®  Guide  process  flow  (Conrow, 
2000;  DAU,  2002a). 

Some  Organizational  and  Behavioral 
Considerations  for  Implementing  Risk 
Management 


Organizational  and  behavioral  consid¬ 
erations  for  implementing  risk  manage¬ 
ment  are  not  discussed  in  the  2000 
PMBOK®  Guide.  A  summary  of  some  key 
considerations  outlined  in  the  DoD  Draft 
Extension  that  apply  to  a  variety  of 
programs  is  now  given. 


Risk  Management 


Risk  Risk  Risk  Risk 

Pianning  Assessment  Handiing  Monitoring 


Risk 

Risk 
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-  Risk  Documentation  - ► 


Figure  1 .  DoD  Risk  Management  Structure  (DAU,  2002a) 
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While  a  comprehensive,  structured, 
repeatable  risk  management  process  is 
important  for  effective  risk  management, 
it  is  equally  important  that  proper  organi¬ 
zational  and  behavioral  considerations 
exist  to  implement  the  process  properly. 
Even  a  state-of-the-art  risk  management 
process  that  is  not  implemented  or  exe¬ 
cuted  well  will  have  a  low  overall  effec¬ 
tiveness. 

While  tailoring  should  be  performed  on 
a  program-to-program  basis,  it  is  impor¬ 
tant  that  risk  management  roles  and 
responsibilities  be  defined  in  the  Risk  Man¬ 
agement  Plan  (RMP)  and  executed  during 
the  program  life  cycle;  else  there  will  be 
little  focus  or  accountability.  Some  key 
roles  and  responsibilities  include  (Conrow, 
2000;  DAU,  2002a;  DAU,  2002b): 

•  Which  group  of  program  personnel  wiU 
have  responsibility  for  risk  management 
decision  making  (e.g..  Risk  Manage¬ 
ment  Board  [RMB]  or  Level  1  Inte¬ 
grated  Product  Team  (IPT)? 

•  Who  chairs  the  RMB  [e.g.,  program 
manager  (PM),  deputy  program  man¬ 
ager  (DPM)]? 

•  Which  group  maintains  and  updates  the 
risk  management  process  (e.g.,  program 
management,  systems  engineering)? 

•  Who  develops  the  RMP  (e.g.,  risk 
manager)? 

•  Which  individual  or  group  is  respon¬ 
sible  for  leading  risk  management 
implementation  and  training  others  in 
risk  management  principles  (e.g.,  risk 
manager)? 


•  Who  performs  risk  management  (e.g., 
everyone)? 

•  Who  assigns  focal  points  to  a  given  risk 
issue  (e.g.,  RMB  or  Level  I  IPT  together 
with  the  cognizant  IPT)? 

•  Who  develops  draft  risk  analyses  (e.g., 
focal  point  together  with  the  cognizant 
IPT  Lead),  and  which  group  approves 
the  results  (e.g.,  RMB  or  Level  1  IPT)? 

•  Who  develops  draft  risk  handling  plans 
(e.g.,  focal  point  together  with  the  cog¬ 
nizant  IPT  Lead),  and  which  group  ap¬ 
proves  the  plans  (e.g.,  RMB  or  Level  1 
IPT)? 

•  Which  group  collects  and  examines 
risk-monitoring  metrics  (e.g.,  focal  point 
and  cognizant  IPT  Lead  together  with 
the  RMB  or  Level  1  IPT)? 

The  above  questions  and  areas  of  re¬ 
sponsibility  are  not  intended  to  be  all- 
inclusive.  In  addition,  the  best  approach 
for  addressing  these  items  will  vary  on  a 
program-to-program  basis  and  depend 
upon  a  host  of  organization,  contractual, 
and  other  considerations. 

While  organizational  roles  and  responsi¬ 
bilities  are  often  relatively  straightforward 
to  develop,  behavioral  considerations 
may  be  much  more  difficult  to  identify 
and  correct  or  enhance  as  warranted. 
While  behavioral  considerations  gener¬ 
ally  involve  creating  an  environment 
conducive  to  performing  risk  manage¬ 
ment  efficiently,  it  often  requires  exam¬ 
ining  attitudes  and  approaches  that  exist 
across  the  program.  Lor  example,  pro¬ 
gram  upper  management  should  not 
only  be  supportive  of  risk  management 
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but  also  use  risk  management  principles 
in  their  decision  making.  This  is  impor¬ 
tant  because  program  personnel  will  ob¬ 
serve  upper  management  involvement  in 
risk  management,  and  if  it  appears  that 
risk  management  is  just  given  lip  ser¬ 
vice,  then  other  program  personnel  may 
be  reluctant  to  participate  actively.  Con¬ 
versely,  this  does  not  mean  that  the  PM 
or  DPM  should  be  the  risk  manager  ex¬ 
cept  possibly  on  small  programs,  but  just 
that  they  are  actively  engaged  and  using 
the  process.  Middle  managers,  such  as 
the  risk  manager,  should  not  lead  risk 
management  because  this  will  potentially 
send  the  wrong  message  of  its  impor¬ 
tance  to  program  personnel.  It  is  equally 
important  to  have  working-level  person¬ 
nel  actively  apply  risk  management  prin¬ 
ciples  in  their  daily  work,  since  they  are 
often  much  closer  to  potential  risk  issues 
than  upper  management.  Thus,  from  the 
PM  to  the  most  junior  program  person¬ 
nel,  risk  management  should  ideally  be 
integrated  as  part  of  the  job  function  — 
not  in  a  bureaucratic  sense,  but  to  assist 
them  in  decision  making. 

The  Performance  Dimension  of 
Consequence  of  Occurrence 


The  2000  PMB  OK®  Guide  includes 
cost,  quality,  schedule,  and  scope  di¬ 
mensions  of  risk  analysis  consequence 
of  occurrence,  whereas  the  DoD  Draft 
Extension  recognizes  the  cost,  perfor¬ 
mance,  and  schedule  (C,P,S)  conse¬ 
quence  of  occurrence  dimensions  (PMI, 
2000;  DAU,  2002a). 

Technical  performance  is  a  concept 
that  is  effectively  absent  from  the  2000 
PMBOK®  Guide,  yet  it  is  a  primary 


driver  for  the  development  of  DoD  as 
well  as  many  non-DoD  and  commer¬ 
cial  systems.  In  fact,  cost  growth  and 
schedule  slippage  often  occur  when  un¬ 
realistically  high  levels  of  performance 
are  required  and  little  flexibility  is  pro¬ 
vided  to  degrade  performance  during 
the  course  of  the  program  (Conrow, 
1995).  Performance  is  also  one  of  the 
three  key  system  attributes  (along  with 
cost  and  schedule)  that  are  used  to  mea¬ 
sure  program  outcomes  (e.g.,  in 
Selected  Acquisition 
Reports). 

Quality  is  often  a 
cause  rather  than  an 
impact  to  the  program 
and  can  generally  be 
broken  down  into  C,P,S 
components  (e.g.,  reli¬ 
ability  given  by  mean 
time  between  failure  is 
often  treated  as  a  perfor¬ 
mance  characteristic). 

(For  example,  holding  all  else  constant, 
poor  quality  will  tend  to  lead  to  in¬ 
creased  cost,  increased  schedule,  and 
decreased  performance  for  a  fabricated 
item.)  Scope  is  not  recommended  as  a 
consequence  of  occurrence  dimension 
since  it  is  often  the  super  set  of  C,P,S, 
and  thus  both  correlated  and  redundant 
with  C,P,S. 

The  Performance  Dimension  of  Monte 
Carlo  Simulation  Modeling 


Monte  Carlo  simulations  are  not  uni¬ 
versally  applicable,  but  should  be  con¬ 
sidered  as  a  candidate  tool  and  tech¬ 
nique  for  conducting  a  risk  analysis. 
The  2000  PMBOK®  Guide  (PMI,  2000) 
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mentions  cost  and  schedule  Monte 
Carlo  simulations,  but  not  performance 
simulations.  Performance  Monte  Carlo 
simulations  are  a  key  architecture,  sys¬ 
tem,  and  component  design  tool  for 
numerous  types  of  defense  and  non¬ 
defense  items.  For  ex¬ 
ample,  the  perfor¬ 
mance  of  an  applica¬ 
tion  specific  integrated 
circuit  (ASIC)  is  often 
modeled  by  a  Monte 
Carlo  simulation  prior 
to  releasing  the  design 
to  silicon  layout,  be¬ 
cause  any  error  in  the 
design  may  lead  to  a  very  costly  and 
time  consuming  redesign.  (In  fact,  for 
a  complex  ASIC  it  is  almost  impossible 
to  verify  the  design  without  using  a 
simulation  or  comparable  tool.)  The 
basic  structure  of  a  performance  simu¬ 
lation  will  depend  upon  the  item  being 
evaluated,  and  will  vary  on  a  case-by¬ 
case  basis;  while  for  cost  and  schedule 
simulations,  the  basic  nature  of  the 
implementation  remains  at  least  some¬ 
what  similar  across  a  wide  variety  of 
projects. 

A  SiRucruRED  Approach  for  Developing  a 
Risk  Handling  Strategy 


While  both  the  2000  PMBOK®  Guide 
(PMI,  2000)  and  DoD  Draft  Extension 
(DAU,  2002a)  employ  the  same  risk  han¬ 
dling  options  (assumption  [acceptance], 
avoidance,  control  [mitigation],  and  trans¬ 
fer  [transference]),  the  DoD  Draft  Exten¬ 
sion  also  emphasizes  a  stmctured  approach 
for  developing  an  overall  risk  handling 
strategy  (Conrow,  2000;  DAU,  2002a). 


(Note:  In  the  2000  PMBOK®  Guide,  the 
risk  handling  step  is  termed  risk  response 
planning.)  Risk  assumption  is  an  acknow¬ 
ledgement  of  the  existence  of  a  particular 
risk  situation  and  a  conscious  decision  to 
accept  the  associated  level  of  risk,  without 
any  special  efforts  to  control  it.  Risk  avoid¬ 
ance  involves  a  change  in  the  concept,  re¬ 
quirements,  specifications  and/or  practices 
that  reduce  risk  to  an  acceptable  level.  Risk 
control  does  not  attempt  to  eliminate  the 
source  of  the  risk  but  seeks  to  mitigate  or 
control  the  risks.  Risk  transfer  often  in¬ 
volves  reallocating  risks  across  design  seg¬ 
ments  (e.g.,  hardware  and  software)  dur¬ 
ing  the  early  development  process  or  re¬ 
distributing  risks  between  parties  (e.g., 
between  buyer  and  seller)  (DAU,  2002b). 

This  structured  approach  includes  first 
selecting  the  best  risk  handling  option,  then 
choosing  the  most  appropriate  implemen¬ 
tation  approach.  This  forms  the  primary 
risk  handling  strategy.  The  structured  ap¬ 
proach  for  choosing  the  most  desirable 
option  and  implementation  approach  is 
important  because  often  program  person¬ 
nel  jump  to  the  “answer”  rather  than  think¬ 
ing  through  whether  or  not  their  strategy 
is  the  most  desirable  one  or  even  an  ac¬ 
ceptable  one.  Eor  high  risks  and  other  cases 
specified  by  the  program  RMB  (or  equiva¬ 
lent),  one  or  more  secondary  risk  handling 
strategies  may  also  be  required.  Here,  pro¬ 
gram  personnel  should  use  the  same  type 
of  process  employed  to  evaluate  the  four 
risk  handling  options  as  in  the  primary  strat¬ 
egy,  choose  the  best  one,  then  select  the 
most  appropriate  implementation  ap¬ 
proach.  While  the  primary  and  secondary 
risk  handling  strategies  may  use  the  same 
risk  handling  option,  they  will  use  a  dif¬ 
ferent  implementation  approach. 


"Risk  control 
does  not  attempt 
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Some  DoD  Draft  Extension 
Enhancements 


Several  enhancements  are  provided  or 
referenced  in  the  DoD  Draft  Extensions. 
The  following  is  a  brief  discussion  of  some 
of  these  enhancements. 

Key  Ground  Rules  and  Assumptions 
AND  Risk  Categories 

The  DoD  Draft  Extension  includes  key 
ground  rules,  assumptions,  and  risk  cat¬ 
egories  as  inputs  to  the  risk  planning  pro¬ 
cess  (DAU,  2002a),  rather  than  as  inputs 
to  risk  identification  (risk  categories)  or 
risk  identification  tools  and  techniques 
(assumptions  analysis)  (PMI,  2000).  The 
advantage  of  the  former  approach  is  that 
the  information  is  included  in  the  RMP 
and  thus  a  consideration  for  every  risk- 
management  process  step,  rather  than 
being  limited  solely  to  risk  identification. 
Eor  example,  including  a  discussion  of 
candidate  risk  categories  as  part  of  the 
RMP  will  help  program  personnel  not  only 
in  risk  identification,  but  also  in  evaluat¬ 
ing  the  level  of  risk  present  (risk  analy¬ 
sis),  and  possibly  in  developing  risk-han¬ 
dling  plans  and  in  monitoring  risk  issues 
(by  focusing  on  potential  risk  issues  bet¬ 
ter).  Even  in  the  case  of  risk  identifica¬ 
tion,  it  is  helpful  for  program  personnel 
to  have  thought  through  potential  ground 
mles  and  assumptions  and  examined  can¬ 
didate  risk  categories  before  performing 
formal  risk  identification  to  reduce  the 
likelihood  that  errors  will  be  made  dur¬ 
ing  this  process. 

Processes  for  Evaluating  Risks 

The  DoD  Draft  Extension  includes  a 
discussion  of  three  different  processes 


commonly  used  for  risk  identification 
and  analysis  (DAU,  2002a).  These  ap¬ 
proaches  include:  critical  process.  Work 
Breakdown  Structure  (WBS),  and  inte¬ 
grated  product/process.  The  2000 
PMB OK®  Guide  includes  a  discussion 
of  the  WBS,  but  not  the  critical  process 
and  integrated  product/process  ap¬ 
proaches  (PMI,  2000).  (The  WBS  ap¬ 
proach  has  strong  historical  precedence 
and  is  widely  used  within  DoD.  In  this 
approach  the  focus  is 
system  elements  and 
associated  with  those 
elements.) 

Critical  process 
approach.  This  ap¬ 
proach  is  used  to  iden¬ 
tify  candidate  techni¬ 
cal  risks  by  assessing 
the  variance  between 
design,  test,  and  pro¬ 
duction  processes  and 
industry  Best  Prac¬ 
tices.  It  was  originally 
applied  to  programs 
transitioning  from  the 
development  to  production  phases.  A 
primary  benefit  of  this  approach  is  that  it 
addresses  common  sources  of  process 
risk  in  many  programs.  The  actual  pro¬ 
gram  baseline  is  developed  and  com¬ 
pared  to  a  baseline  of  industrywide  pro¬ 
cesses  and  practices  that  are  critical  to 
the  program.  This  forms  a  basis  for  per¬ 
forming  risk  identification  and  assist¬ 
ing  in  performing  risk  analysis  and  to 
some  extent  selecting  implementation 
approaches  for  risk  handling.  The  criti¬ 
cal  process  approach  has  many  benefits, 
but  these  processes  are  often  not  directly 
related  to  individual  WBS  elements.  It 
may  also  be  difficult  to  implement  early 


on  products  or 
the  processes 


"...[T]hree 
different  pro¬ 
cesses  commonly 
used  for  risk 
identification  and 
analysis  •••  include: 
critical  process. 
Work  Breakdown 
Structure  (WBS), 
and  integrated 
preduct/precess." 
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in  the  development  process  before  ma¬ 
ture  design,  test,  production,  and  other 
processes  are  in  place. 

Integrated  Process/Product  ap¬ 
proach.  This  technical  risk  assessment 
approach  is  derived  primarily  from  the 
critical  process  approach  and  incorpo¬ 
rates  some  aspects  of  the  WBS  approach. 
Here,  the  systems  engineering  process 
defines  design  and  product  solutions  in 
terms  of  design,  test,  and  manufactur¬ 
ing  requirements.  The  resulting  solutions 
and  their  associated  processes  are  com¬ 
pared  to  a  baseline  of  industrywide  pro¬ 
cesses  and  practices  that  are  critical  to 
the  program.  These  requirements  are  also 
mapped  to  the  WBS,  which  provides  a 
stmcture  for  relating  the  program’s  tech¬ 
nical  objectives  to  product-oriented  ele¬ 
ments  and  the  processes  needed  for  their 
achievement.  The  emphasis  in  this  ap¬ 
proach  is  on  systems  engineering  along 


with  process  and  product  solutions,  which 
is  particularly  important  during  the  initial 
phases  of  product  design  and  produc¬ 
tion. 

Qualitative  Risk  Analysis  Tools  and 
Techniques 

The  use  of  ordinal  probability  and  con¬ 
sequence  of  occurrence  scales  is  a  tool  and 
technique  common  to  the  2000  PMBOK® 
Guide  (PMI,  2000)  and  the  DoD  Draft 
Extension  (DAU,  2002a),  the  DoD  Draft 
Extension  cautions  not  to  perform  math¬ 
ematical  operations  on  results  obtained 
from  (raw)  ordinal  risk  scales.  Instead,  it 
recommends  mapping  the  probability  of 
occurrence  and  consequence  of  occur¬ 
rence  results  into  risk  levels  by  using  a  risk 
mapping  matrix.  (An  example  risk  map¬ 
ping  matrix  is  given  in  Eigure  2.  Here,  or¬ 
dinal  probability  and  consequence  levels 
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Figure  2.  Example  Risk  Mapping  Matrix 
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are  assumed  and  E  >  D  >  C.  The  result¬ 
ing  risk  levels  are  low  [L],  medium  [M], 
and  high  [H].) 

This  is  because  most  risk  scales  have 
coefficients  that  are  ordinal,  not  cardi¬ 
nal,  and  their  true  value  is  unknown. 
(Note:  Cardinal  values  can  he  presented 
as  ordinal  values  [e.g.,  0.5  and  0.4  as  E 
and  D  where  E  >  D],  hut  ordinal  values 
should  never  he  presented  as  or  assumed 
to  he  cardinal  values  [e.g.,  E  and  D  where 
E  >  D  as  0.5  and  0.4].)  Performing  math¬ 
ematical  operations  on  results  obtained 
from  ordinal  scales  can  lead  to  results 
that  will  at  best  be  uncertain  or  mislead¬ 
ing,  if  not  completely  meaningless,  and 
could  result  in  erroneous  risk  ratings 
even  in  terms  of  the  Top  5  program  risks 
(Conrow,  2000;  DAU,  2002b).  It  can  be 
easily  shown  that  errors  exceeding  sev¬ 
eral  hundred  percent  can  exist  in  ordinal 
scale  coefficients  that  are  erroneously 


assumed  to  be  cardinal.  Such  errors  may 
overwhelm  the  accuracy  and  certainty 
of  most  all  risk  analyses.  While  proce¬ 
dures  exist  to  calibrate  ordinal  scales, 
they  are  generally  difficult  and  costly 
to  implement. 

The  DoD  Draft  Extension  also  cau¬ 
tions  against  the  use  of  a  risk-mapping 
matrix  that  has  asymmetric  boundaries 
between  risk  levels  (e.g.,  E,  M,  H).  This 
is  because  the  risk  level  boundaries  are 
often  guessed  and  not  based  upon  any 
conclusive  evidence.  In  addition,  an 
asymmetric  risk-mapping  matrix,  which 
requires  either  a  risk  averse  or  risk  taker 
position,  cannot  be  used  at  the  same 
time  as  other  methodologies,  such  as 
expected  monetary  value  (equivalent 
to  deterministic  decision  tree  analy¬ 
sis),  which  requires  a  risk  neutral 
assumption. 


Dr.  Edmund  H.  Conrow  is  a  risk  management  consultant  to 
government  and  industry  with  more  than  20  years  experience.  He 
has  helped  develop  much  of  the  Department  of  Defenses  best 
practices  on  risk  management  and  has  also  served  as  a  risk  manager 
on  a  variety  of  programs.  Conrow  is  the  author  of  the  book  Effective 
Risk  Management:  Some  Keys  to  Success,  American  Institute  of 
Aeronautics  and  Astronautics.  He  holds  Ph.D.s  in  both  general 
engineering  and  policy  analysis. 
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